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Purpose of the guide 


Although high school graduation rates continue to rise in the United States, reaching 81 percent in the 
2012-2013 school year (U.S. Department of Education, 2015), dropout remains a pervasive issue for 
education systems across the nation. In recent years, Early Warning Systems (EWS), which utilize 
administrative data to identify students at risk of dropout and help target interventions to help students 
graduate, have become increasingly popular among districts. At its core, an EWS can provide direction 
for teams of education administrators, researchers, counselors, parents and community members working 
together to help students stay in school. 1 

For districts without an EWS in place, this guide is intended to focus conversations and decisions about 
whether the district is ready for an EWS and when and how to proceed with EWS implementation if the 
district is ready. The discussion guide will help readers assess the degree to which each of four readiness 
signs exist within their districts and data systems. If these signs are present, districts may be well- 
positioned to successfully implement an EWS. Otherwise, completing the guide can help readers identify 
additional information and next steps their districts can take to enhance their readiness. 

Who should use the guide 

The guide is designed to be flexible and accommodate districts of varying size and structure, but may be 
most useful to small- or mid-sized districts that have not had an EWS in the past and are exploring 
whether or not to implement an EWS to address a dropout problem. EWS teams can be composed of 
stakeholders from the district or school level, or a combination of both, and can include district 
administrators, researchers, information technology staff, counselors, students, parents, and community 
representatives, depending on the needs of the district, school, or community. Whatever the composition 
of the group, there should be a core EWS team that includes key decisionmakers who are committed to 
taking part in each discussion in this guide. 2 


How to use the Discussion Guide 

The guide provides a framework for facilitating four discussions—one for each of the four EWS readiness 
signs. For each discussion there is a one to two-page handout (pages 6-13) that includes references to 
recommended readings and a series of guiding questions that will be used to guide the discussion. At the 
end of each discussion, participants are asked to rate their confidence as a group that the “readiness sign” 
is present within the district and list the key conclusions from their discussion that justify their rating. 
There is no established “cut point” on this confidence scale that the district must meet in order to be 
deemed “ready.” Instead, these ratings are designed to provide a quick snapshot of district readiness in 
that area to inform decisionmaking. 


1 For more information on EWS in general, see Bruce, M., Bridgeland, J. M., Fox, J. H., & Balfanz, R. (2011). On 
Track for Success: The Use of Early Warning Indicator and Intervention Systems to Build a Grad Nation. 
Washington, DC: Civic Enterprises. Retrieved from 

http://www.civicenterprises.net/MediaLibrary/Docs/on track for success.pdf 

2 For more information on creating an EWS team, see Frazelle, S. & Nagel, A. (2015). A practitioner’s guide to 
implementing early warning systems (REL 2015-056). Washington, DC: U.S. Department of Education, Institute of 
Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational 
Laboratory Northwest. Retrieved from http://ies.ed.gov/ncee/edlabs/proiects/proiect.asp?ProiectID=396 
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There are no limits to the number of participants to include in each discussion. With the exception of the 
core team, participants may differ from one discussion to the next, depending upon which district or 
school level staff are best equipped to answer the discussion questions and assess the district’s status on 
each readiness sign. For some districts it might make sense to convene a single, two-hour meeting in 
which the same group of participants would conduct all four discussions at once. For other districts, the 
meetings may involve different participants and be scheduled separately over a multi-week period. 

Follow the five steps below to facilitate each readiness discussion: 

Step 1: Designate a facilitator who will be responsible for scheduling the readiness discussion, 

leading the participants through the discussion questions, and taking notes. This 
individual does not have to be an expert in the discussion topic, but should have 
experience in meeting facilitation, and some basic knowledge of an EWS in order to 
drive follow-up questions as needed. 

Step 2: Identify and recruit participants. As you create your list of participants, refer to each 

readiness discussion (pages 6, 8, 10 and 12) to determine what areas of expertise should 
be represented in specific discussions related to participant’s expertise. Then, identify 
specific individuals who have expertise in one or more of those areas to be invited to the 
discussion. For example, for the readiness discussion on Tiered Interventions, you may 
decide you will need to invite individuals with on-the-ground experience in providing 
student interventions, familiarity with the current process for identifying students in need 
of interventions, knowledge of effective intervention strategies, etc., and then ask one or 
more school guidance counselors and intervention specialists to take part in the 
discussion. 

Step 3: Share the discussion sheet (pages 5-12) with participants at least one week in advance 

of the readiness discussion so they have time to review the recommended readings and 
discussion questions and come prepared for an engaging discussion on the district’s 
readiness for an EWS. 

Step 4: Facilitate the discussion. Begin the readiness discussion with a review of the 

“Considerations” section and then move through the discussion questions, allowing time 
for all the individuals who participate to share insights and draw upon what they learned 
from the recommended readings. You should allow at least thirty minutes for each 
readiness discussion. 

Step 5: Summarize the conclusions of the readiness discussion by asking the group to agree on 

a confidence rating regarding whether the district displays the sign of readiness that was 
discussed. Provide a written rationale for the confidence rating by summarizing key 
takeaways from the discussion. 
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Four signs your district is ready for an Early Warning System 


Researchers at Regional Educational Laboratory (REL) Pacific reviewed the research literature regarding 
implementation of EWS and identified four organizational conditions, or “readiness signs”, that promote 
successful implementation. 


(1) A clear vision for addressing the dropout problem'. District leaders along with participants who 
have been identified to be part of the readiness discussions have defined the nature and 
underlying causes of the district’s dropout problem including the prevalence of dropout in the 
district, whether dropout rates are rising or falling over time and how the risk of dropout differs 
among student subgroups. District leaders and EWS teams are also ready to investigate the root 
causes of the dropout problem and articulate a clear and shared vision that includes specific goals 
for improving graduation rates and a coherent strategy for how an EWS would support those 
goals (Therriault et al., 2013; Jerald, 2006). 

(2) A high quality data system : The district has the data management tools and technical capacity to 
track student outcomes over time. This includes the ability to collect, clean, verify, organize, and 
link longitudinal student-level records from multiple data sources, including attendance data, 
discipline records, report cards, and graduation outcomes (Bruce et al., 2011; Frazelle & Nagel, 
2015). 

(3) Access to expertise in data analytics: The district has access to an individual or partner who can 
conduct the statistical calculations necessary to identify or validate a set of 9 th grade indicators 
that accurately predict high school dropout for its students (Heppen & Therriault, 2008; Stuit et 
al., 2016). 

(4) Capacity to deliver a tiered intervention system : The district has the resources and capacity to 
implement a tiered intervention system that delivers schoolwide, targeted (small group), and 
individualized interventions that are differentiated based on the specific needs of students 
identified as at risk by the EWS and the ability to track the implementation and effectiveness of 
these interventions over time (Therriault et al., 2013; Frazelle & Nagel, 2015). 
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Discussion 1: Do we have a clear vision for 
reducing high school dropout rates? 

Considerations 

A clear written vision statement will help motivate and guide the work of 
everyone involved in implementing the EWS (Waters, Marzano, & 

Mcnulty, 2003; Elmore, 2003). A clear vision statement for an EWS will have three defining features: 


(1) It will be based on a thorough understanding of the nature of 
the dropout problem, including knowledge of how prevalent 
dropout is within the district, whether dropout rates are rising 
or falling over time, how the risk of dropout differs across 
student subgroups, and readiness for an investigation of the 
root causes of the dropout problem. 

(2) It will include concrete goals for reducing dropout rates and 
improving graduate rates. These goals should be written and 
articulate specific improvement targets and a timeline for 
meeting them. 

(3) It will detail the strategy for how an EWS will help the district 
achieve these goals. 


One way to ensure that your district has a clear vision for an EWS 
and is able to monitor progress towards reducing dropout is to build a logic model. A logic model can 
describe how the resources and activities involved in implementing an EWS are expected to impact short- 
, medium- and long-term student outcomes. For interactive applications to build logic models and 
evaluate outcomes, link to http://relpacific.mcrel.org/resources/elm-app/. 


Recommended Readings 

On Track for Success: The Use of Early 

Warning Indicator and Intervention Systems 

to Build a Grad Nation 
Civic Enterprises & The Everyone 
Graduates Center at Johns Hopkins 
University (Particularly pp. 9-19) 

Vision, Leadership, and Change 
SEDL Issues about Change. Volume 2, 3. 

Logic Models: A Tool for 
Designing and Monitoring Program 

Evaluations 

REL Pacific 



Discussion Questions 

(1) What do we know about the dropout problem in our district? What are our district’s cohort 
graduation and dropout rates? How are dropout rates trending over time? What students are most 
likely to drop out and why? (Problem Definition) 

(2) What are our specific goals for reducing high school dropout rates in the future? What 
improvement do we expect to see and by when? Are these goals realistic? (Goal Clarity) 

(3) What is our vision for how an EWS will help us meet our goals for reducing dropout rates? How 
would an EWS help us in setting new goals in this area over time? (Strategic Vision) 

(4) What additional information would we need to gather or what additional actions would we need 
to take in order to improve our readiness in this area? 
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Conclusions 


Based on this discussion, how confident are you that the district has a clear vision for how an EWS will 
be used to reduce dropout rates and improve graduation rates? (Circle One) 

Not at all confident Somewhat confident Confident Very Confident 

Please explain your response. What takeaways from the discussion led you to assign the above 
confidence rating? 
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Discussion 2: Do we have the data system required 
to build a set of early warning indicators and 
quickly identify students who are at risk of 
dropout? 

Considerations 

Your data management system will need to perform two functions in order to effectively support an EWS: 


(1) Indicator Selection: Produce a longitudinal student-level data set 
that will be used for the statistical analysis required to identify 
accurate early warning indicators. This data set should include 
student-level records for all students who enrolled as incoming 
freshmen for the first time for two cohorts of students whose 
expected graduation dates have passed (e.g., the freshmen cohort 
that entered in the fall of 2008 and was expected to graduate in 
the spring of 2012). 

At a minimum, this data should include 9 th grade attendance, 
behavior, and coursework records, which the literature suggests 
as a starting point for indicators, along with demographic 
information and a way to link these data to the students’ 
graduation or dropout status and verify the validity of the data. 

See the recommended readings for more information. 

If you do not currently have the longitudinal data necessary to 
validate indicators for your district, you can consider adopting 
indicators and thresholds that have been used by other districts, and then adjust these indicators and 
thresholds in the future when you have sufficient local data. For examples of indicators and 
thresholds used by other districts, see What Matters for Staying On-track and Graduating in Chicago 
Public High Schools and pages 5-8 of REL Northwest’s A Practitioner's Guide to Implementing 
Early Warning Systems , two of the recommended readings. 

(2) Risk Classification: Be capable of applying the early warning indicators to fall semester data on 
currently enrolled 9 th grade students so that the students who are at risk of dropout can be identified 
and practitioners, parents, and students can be notified of their at-risk status by early in the spring 
semester of 9 th grade at the latest. For example, if an early warning indicator classifies all students 
with attendance rates below 90 percent as at risk, the data system must be able to access the fall 
semester attendance data and assign flags to all students who fall below the 90 percent threshold in 
time to place these students in interventions for the spring semester. The earlier the system can 
identify students, the better. 


Recommended Readings 

A Practitioner's Guide to Implementing 

Early Warning Systems 

REL Northwest 

(Particularly “Identifying accurate 
indicators”, starting on p. 5) 

Traveling Through Time: The Forum 

Guide to Longitudinal Data Systems 

National Forum on Education Statistics 

What Matters for Staving On-track and 

Graduating in Chicago Public High 

Schools 

Consortium on Chicago School Research 
at the University of Chicago 




Readiness 
Sign #2 

High quality data 
system 
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Discussion Questions 

(1) Do we have the capacity to construct a historical longitudinal data file that can be used to identify 
accurate 9 th grade predictors of dropout? If not, do we have the ability to create a longitudinal 
data file to allow us to track current 9 th graders going forward? (Indicator Selection) 


(2) Do we have the capacity to apply the early warning indicators to data on currently enrolled 
students and quickly identify students who are classified as at risk based on those indicators? If 
so, how quickly? What tools, such as a dashboard or spreadsheet, are we using to identify 
students? (Risk Classification) 

(3) What additional information would we need to gather or what additional actions would we need 
to take in order to improve our readiness in this area? 

Conclusions 

Based on this discussion, how confident are you that the district has the necessary data systems in place to 
be able to identify predictive indicators of dropout for 9 th grade students and track student results on those 
indicators? (Circle One) 

Not at all confident Somewhat confident Confident Very Confident 

Please explain your response. What takeaways from the discussion led you to assign the above confidence 
rating? 
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Discussion 3: Do we have the capacity to analyze our 
data and identify a set of 9 th grade early warning 
indicators that accurately predict dropout? 


Considerations 

To implement an EWS, your district must establish a set of 9 th 
grade early warning indicators that accurately predict which students 
will drop out of high school. While research shows that the “ABC” 
indicators (attendance, behavior, and coursework) are consistently 
accurate predictors of dropout for students in multiple contexts 
(Hartman, Wilkins, Gregory, Gould, & D’Souza, 2011; Norbury, et 
al., 2012), it is important to verify that this is the case for your 
particular student population. If your indicators are not accurate, the 
EWS will not be able to identify the students who are truly at risk of 
dropout and most in need of support. 

There are several methods to identify accurate early warning 
indicators available to you and your team. Two commonly used 
approaches are the CART method, which is explained further in the 
report and infographic by REL Southeast and logistic regression, 
which is explained in more detail in Appendix B of the REL Midwest 
report. See the recommended readings for more about these methods 
and how they have been used to develop EWS indicators. If your 
district does not have the capacity to identify accurate EWS 
indicators, two additional options are to seek external partners that 
can help identify and validate EWS indicators, or determine whether 
your state or other districts within your state have an EWS tool that 
could be accessed. If your state or another district within your state does have an EWS tool, it will be 
important to determine whether the students used to verify those specific indicators and cut points were 
similar or different from students in your district. If the students used to verify those indicators were 
different, it is possible that those indicators or cut points may not be accurate for your district. 


Recommended Readings 

Developing Early Warning Systems to 

Identify Potential High School Dropouts 

National High School Center 

Comparing Methodologies for Developing 
an Early Warning System: Report and 
Infographic 
REL Southeast 

Identifying Potential Dropouts: Key 

Lessons for Building an Early Warning 

Data System 

Achieve, Inc. American Diploma Project 
Network 

Identifying Early Warning Indicators in 
Three Ohio School Districts: Report and 
Infographic 
REL Midwest 













Discussion Questions 

(1) Does our staff have the capacity to conduct the analysis that is required to identify a set of early 
warning indicators that accurately predict dropout for our students? (Internal Analytic Capacity) 

If your staff does have this capacity: 

a. What individuals or departments have this capacity? 

If your staff does not have this capacity: 

b. Does our state or other districts within our state offer an EWS tool that we could access? 
If so, are the indicators and cut points used in that system verified to be accurate 
predictors of dropout for our particular students? 

c. Do we know of any external organizations that could partner with us to help identify and 
validate our indicators and if necessary modify our EWS? 

(2) What additional information would we need to gather or what additional actions would we need 
to take in order to improve our readiness in this area? 


Conclusions 

Based on this discussion, how confident are you that the district has access to an individual or partner 
with the analytic skills to identify and validate a set of 9 th grade indicators that accurately predict high 
school dropout in your district? 

Not at all confident Somewhat confident Confident Very Confident 

Please explain your response. What takeaways from the discussion led you to assign the above 
confidence rating? 
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Discussion 4: Are we prepared to deliver a tiered 
intervention system that provides differentiated 
support to the students who are identified as at risk of 
dropout by the early warning system? 

Considerations 

Once students are identified as at risk of dropping out, they need to be matched with effective 
interventions that address their specific risk factors. As defined in the recommended reading, A 
Practitioner’s Guide to Implementing Early Warning Systems, 
interventions are “programs and policies.. .in place to help students 
succeed in school” (p. 10). For example, if the district uses attendance, 
behavior, and coursework indicators to predict which students are at risk 
of dropout, the district will need to make interventions available that can 
address each of those factors. 

The Institute of Education Sciences’ What Works Clearinghouse (WWC) 
is a good resource for identifying interventions for a variety of student 
needs. The WWC reviews the existing research on different programs, 
products, practices, and policies in education. Its goal is to provide 
educators with the information they need from high-quality research to 
make evidence-based decisions, such as whether to adopt a specific type 
of program or policy. 


Fully implementing an EWS includes the use of a tiered intervention system, with students who are most 
at risk of dropping out receiving additional layers of support. School-wide interventions are used to 
address risk factors that are systemic for most or all students in the school. Targeted interventions are 
delivered to a subset of the entire student population (approximately 15-20 percent of students in a school 
or a specific grade level). Intensive interventions are provided to the students who are most severely at 
risk of dropout (approximately 5-10 percent of students in a school or a specific grade level) and 
commonly occur in small-group or one-on-one settings. Without the capacity to deliver a tiered 
intervention system directed at specific student needs, the district will not be able to use EWS information 
for its intended purpose: getting at-risk students back on track to graduate. 

Districts also need to be able to evaluate the implementation and effectiveness of their dropout 
interventions in order to improve their offerings over time. Districts should track which interventions each 
student receives and how their performance on the EWS indicators changes over time. By looking at this 
information across students and student subgroups, the EWS team can monitor trends in the effectiveness 
of interventions. While these trends do not provide a definitive answer as to whether or not student 
improvement was driven by participation in particular interventions, they can provide some indication of 
whether interventions are meeting the needs of students and help districts make adjustments to the 
interventions they offer. For districts that wish to perform their own in-depth evaluation of an 
intervention, RCT-YES is a free, user-friendly tool that can help districts conduct a more rigorous 
assessment of the intervention’s impact. 


Recommended Readings 

A Practitioner's Guide to 
Implementing Early Warning 

Systems 

REL Northwest 

(Particularly pp. 10-14) 

High School Early Warning 
Intervention Monitoring System 
Implementation Guide National High 
School Center at the American 
Institutes for Research 
(Particularly pp. 21-24) 
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Discussion Questions 

(1) What interventions do we currently have in place to address risk factors identified through 
potential dropout indicators such as attendance, behavior, or coursework? Do we have the 
capacity to identify and implement multiple evidence-based intervention strategies at the school, 
targeted (small group), and individualized levels that are differentiated based on the needs of 
students identified as at risk by the EWS? (Dropout Prevention Resources) 

(2) What do we know about the use of multi-tiered intervention systems and how they are used 
within EWS? What do we need to learn more about? (Understanding of Tiered Interventions) 

(3) Do we have the capacity to monitor and evaluate the implementation and effectiveness of our 
intervention strategies and act on that information? (Ability to Monitor Effectiveness of 
Interventions) 

(4) What additional information would we need to gather or what additional actions would we need 
to take in order to improve our readiness in this area? 


Conclusions 

Based on this discussion, how confident are you that district has the capacity to implement a tiered 
intervention system that addresses the different needs of all students who are identified as at risk of 
dropout by the EWS? (Circle One) 

Not at all confident Somewhat confident Confident Very Confident 

Please explain your response. What takeaways from the discussion led you to assign the above 
confidence rating? 
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Next Steps 


Based on the discussions in this guide, EWS teams should now have a better understanding of whether 
their districts have a clear vision for addressing the dropout problem, a high-quality data system and 
access to expertise in data analytics that will allow them to accurately predict which students are at risk of 
dropping out, and the capacity to deliver a tiered intervention system to meet the needs of students. 

The next step is to review the conclusions from the four discussions and prepare to debrief key 
stakeholders on the district’s overall readiness for an EWS. 


Step 1: Summarize your team’s findings from across the four discussions, including the 

confidence ratings that your team agreed upon regarding each readiness sign. 

Step 2: List and describe the action steps that the district would need to take to improve its 

readiness in each of the areas discussed. You may have found that there are very few 
action steps needed in relation to some areas while others need more attention. You may 
have action steps to take even if you found that your team was “Very Confident” that the 
readiness sign was present in your district. 

Step 3: Develop a timeline for accomplishing these action steps that shows how long it would 

take to implement each step and an overall estimate of how long it would take for the 
district to develop each of the four signs of readiness. 

Step 4: Meet with key stakeholders, such as the district superintendent to report on your findings, 

action steps, and timeline and assess the interest in moving forward with pursuing an 
EWS. 

If a district is not yet ready to implement an EWS, but is still interested in pursuing one in the future, it 
can further refine the action steps and timeline that its team developed and create a plan for achieving 
them. Drawing on the “Recommended Readings” can help districts refine their plans. 

If a district is ready to implement an EWS, there are a number of resources that can help it get started. 
Two that provide detailed information to guide districts through the process are A Practitioner's Guide to 
Implementing Early Warning Systems and the High School Early Warning Intervention Monitoring 
System Implementation Guide . The readiness discussions may have identified opportunities to enhance 
your district’s capacity to implement a high-quality EWS, and you can use these findings to inform your 
implementation efforts. 

Whether a district is ready to implement an EWS in the near-term or needs to further develop its readiness 
before putting an EWS in place, the Early Warning Systems Learning Series provided by the Regional 
Educational Laboratory Program has several resources that can help you learn more about EWS, on topics 
ranging from Developing a Culture of Data Use to Maximize the Use of an Early Warning System to 
Considering the Best EWS Model to Fit Your Needs to Going Districtwide: Implementation Lessons for 
Using Early Warning Systems and Local Risk Indicators . 
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